Data challenge - churn prevention for Revolut¶
Prepare packages¶
We're reading below standard data science toolkit, including tools for data processing (e.g. dask - "pandas in parallel"), visualization tools (like e.g. plotly, seaborn) and some machine-learning frameworks (scikit-learn, and similar).
System packages¶
In [1]:
import multiprocessing
In [2]:
import sys
import os
In [3]:
from datetime import datetime
In [4]:
import random
In [5]:
module_path = os.path.abspath(os.path.join('../code/'))
if module_path not in sys.path:
sys.path.append(module_path)
Data science toolkit¶
In [6]:
import numpy as np
import pandas as pd
import dask.dataframe as dd
Machine learning¶
In [7]:
import featuretools as ft
In [8]:
from sklearn.linear_model import LogisticRegression
In [9]:
from sklearn.ensemble import RandomForestClassifier
In [10]:
from sklearn.ensemble import GradientBoostingClassifier
In [11]:
from sklearn.model_selection import train_test_split
In [12]:
from sklearn.metrics import classification_report
In [13]:
import shap
Visualizations¶
In [14]:
import seaborn as sns
In [15]:
import plotly.express as px
Other¶
In [16]:
from pandas_profiling import ProfileReport
In [17]:
from rnormalize import normalize_user_id
In [18]:
from rtools import get_user_plan
In [19]:
from dateutil import parser
Read data¶
This Jupyter notebook follows rules of reproducible resarch, meaning it can be re-run (after preparing the environment, as defined in requirements.txt) - resulting in similar values in the output and exactly same business conclussions. The only requirement is to have the same file/dir structure - notebook should be located in the notebooks directory, above, we should have data in the data directory, and some additional sources in the code directory.
In [20]:
random.seed(42)
file location¶
In [21]:
devices_filename = '../data/rev-devices.csv'
notifications_filename = '../data/rev-notifications.csv'
users_filename = '../data/rev-users.csv'
transactions_filename = '../data/rev-transactions.csv'
read csv files¶
In [22]:
devices = dd.read_csv(devices_filename)
f'File {devices_filename} was successfully read'
Out[22]:
In [23]:
notifications = dd.read_csv(notifications_filename)
f'File {notifications_filename} was successfully read'
Out[23]:
In [24]:
users = dd.read_csv(users_filename)
f'File {users_filename} was successfully read'
Out[24]:
In [25]:
transactions = dd.read_csv(transactions_filename)
f'File {transactions_filename} was successfully read'
Out[25]:
Data book¶
The objective of the Data Book is to provide useful summaries of data, which are easily understood by everyone.
table: devices.csv
a table of devices associated with a user
- brand: string corresponding to the phone brand
- user_id: string uniquely identifying the user
table: users.csv
a table of user data
- user_id: string uniquely identifying the user
- birth_year: integer corresponding to the user’s birth year
- country: two letter string corresponding to the user’s country of residence
- city: two string corresponding to the user’s city of residence
- created_date: datetime corresponding to the user’s created date
- user_settings_crypto_unlocked: integer indicating if the user has unlocked the crypto currencies in the app
- plan: string indicating on which plan the user is on
- attributes_notifications_marketing_push: float indicating if the user has accepted to receive marketing push notifications
- attributes_notifications_marketing_email: float indicating if the user has accepted to receive marketing email notifications
- num_contacts: integer corresponding to the number of contacts the user has on Revolut
- num_referrals: integer corresponding to the number of users referred by the selected user
- num_successful_referrals: integer corresponding to the number of users successfully referred by the selected user (successfully means users who have actually installed the app and are able to use the product)
table: notifications.csv
a table of notifications that a user has received
- reason: string indicating the purpose of the notification
- channel: string indicating how the user has been notified
- status: string indicating the status of the notification
- user_id: string uniquely identifying the user
- created_date: datetime indicating when the notification has been sent
table: transactions.csv
a table with transactions that a user made
- transaction_id: string uniquely identifying the transaction
- transactions_type: string indicating the type of the transaction
- transactions_currency: string indicating the currency of the transaction
- amount_usd: float corresponding to the transaction amount in USD
- transactions_state: string indicating the state of a transaction
- COMPLETED - the transaction was completed and the user's balance was changed
- DECLINED/FAILED - the transaction was declined for some reason, usually pertains to insufficient balance
- REVERTED - the associated transaction was completed first but was then rolled back later in time potentially due to customer reaching out to Revolut
- ea_cardholderpresence: string indicating if the card holder was present when the transaction happened
- ea_merchant_mcc: float corresponding to the Merchant Category Code (MCC)
- ea_merchant_city: string corresponding to the merchant’s city
- ea_merchant_country: string corresponding to the merchant’s country
- direction: string indicating the direction of the transaction
- user_id: string uniquely identifying the user
- created_date: datetime corresponding to the transaction’s created date
First glance at the data¶
We're checking here what's the size of data and how many columns we got
Data volume¶
In [26]:
f'There are {devices.shape[0].compute()} rows in the "devices" table'
Out[26]:
In [27]:
f'There are {notifications.shape[0].compute()} rows in the "notifications" table'
Out[27]:
In [28]:
f'There are {users.shape[0].compute()} rows in the "users" table'
Out[28]:
In [29]:
print(f'''There are {transactions.shape[0].compute()} rows in the "transactions" table.
The transactions table is {int(transactions.shape[0].compute() / users.shape[0].compute())} times bigger
than the number of users.''')
Columns and their data types¶
In [30]:
devices.dtypes
Out[30]:
This is rather self-explanatory. It tells what type of device user is using (column - brand).
In [31]:
notifications.dtypes
Out[31]:
This is an interesting table. We're especially thrilling to investigate what type of messages are sent to the users (column - reason). Probably they're mostly "push" messages.
In [32]:
users.dtypes
Out[32]:
Data on the user. We have information on the type of plan, as well as, some extra data about number of marketing messages.
In [33]:
transactions.dtypes
Out[33]:
Transactional data. It should be possible to find transaction type (atm/transfer/online), as well as country of the merchant.
Data profiling¶
Users¶
In [34]:
users_df_profile = ProfileReport(users.sample(frac=0.25).compute(),
progress_bar=False,
title='Pandas Profiling Report ("Users" dataframe)',
html={'style':{'full_width':True}})
In [35]:
users_df_profile.to_notebook_iframe() # .to_widgets()
In [36]:
users_df_profile.to_file(output_file="out/users_df_profile_p0.25.html") # saving the whole report to an html file
Devices¶
In [37]:
devices_df_profile = ProfileReport(devices.sample(frac=0.25).compute(),
progress_bar=False,
title='Pandas Profiling Report ("Devices" dataframe)',
html={'style':{'full_width':True}})
In [38]:
devices_df_profile.to_notebook_iframe() # .to_widgets()
In [39]:
devices_df_profile.to_file(output_file="out/devices_df_profile_p0.25.html")
Notifications¶
In [40]:
notifications_df_profile = ProfileReport(notifications.sample(frac=0.1).compute(),
progress_bar=False,
title='Pandas Profiling Report ("Notifications" dataframe)',
html={'style':{'full_width':True}})
In [41]:
notifications_df_profile.to_notebook_iframe() # .to_widgets()
In [42]:
notifications_df_profile.to_file(output_file="out/notifications_df_profile_p0.10.html")
Transactions¶
In [43]:
transactions_pd = transactions.compute() # store df in memory
In [44]:
transactions_df_profile = ProfileReport(transactions_pd,
minimal=True,
pool_size=multiprocessing.cpu_count()-1,
title='Pandas Profiling Report ("Transactions" dataframe)',
html={'style':{'full_width':True}})
In [45]:
transactions_df_profile.to_notebook_iframe() # .to_widgets()
In [46]:
transactions_df_profile.to_file(output_file="out/transactions_df_profile_pALL.html")
Data engineering¶
In [47]:
users_pd = users.compute()
users_pd['user_id'] = users_pd['user_id'].apply(lambda x: normalize_user_id(x)).astype(int)
In [48]:
devices_pd = devices.compute().reset_index().rename(columns={'index': 'device_id'})
devices_pd['user_id'] = devices_pd['user_id'].apply(lambda x: normalize_user_id(x)).astype(int)
In [49]:
notifications_pd = notifications.compute().reset_index().rename(columns={'index': 'notification_id'})
notifications_pd['user_id'] = notifications_pd['user_id'].apply(lambda x: normalize_user_id(x)).astype(int)
In [50]:
transactions_pd['user_id'] = transactions_pd['user_id'].apply(lambda x: normalize_user_id(x)).astype(int)
In [51]:
transactions_pd.memory_usage(deep=True)
Out[51]:
Exploratory data analysis¶
Data understanding¶
Let's vizualize some data!
Understanding notifications¶
In [52]:
notifications_pd.reason.value_counts()
Out[52]:
In [53]:
notifications_pd.channel.value_counts()
Out[53]:
The 'SMS' channel is used quite rarely, so we're going to omit it in our visualizations.
In [54]:
notifications_plot_data = notifications_pd[notifications_pd.channel != 'SMS'].sample(n=30*1000)
notifications_plot_data['user_plan'] = notifications_pd['user_id'].apply(lambda x: get_user_plan(x, users_pd))
notifications_plot_data['total_notifications'] = 1
In [55]:
agg_notifications_plot_data = notifications_plot_data.groupby(['reason',
'channel',
'status',
'user_plan']).agg({'total_notifications': 'sum'})
In [56]:
fig = px.bar(agg_notifications_plot_data.reset_index(),
x="reason",
y='total_notifications',
color="user_plan",
barmode="stack",
facet_row="channel", facet_col="status")
fig.show()
Understanding transactions¶
In [57]:
transactions_pd
Out[57]:
In [58]:
transactions_pd.ea_merchant_country.value_counts()
Out[58]:
In [59]:
transactions_for_plot = transactions_pd.sample(n=10*1000)
transactions_for_plot['amount_usd'] = transactions_for_plot['amount_usd'].apply(lambda x: np.log(x))
In [60]:
g = sns.catplot(x="direction",
y="amount_usd",
hue="transactions_type",
col="transactions_state",
data=transactions_for_plot, kind="strip",
height=4, aspect=0.5)
Time range of transactional data¶
In [61]:
print(f'First transaction in the data is from {transactions_pd.created_date.min()}')
print(f' and the last one is from {transactions_pd.created_date.max()}')
max_date = transactions_pd.created_date.max()
In [62]:
max_date_dt = parser.parse(max_date)
max_date_dt
Out[62]:
Feature engineering¶
Define relations¶
In [63]:
entities = {"users" : (users_pd, "user_id"),
"devices" : (devices_pd, "device_id"), # we really need that index-as-column
# https://github.com/FeatureLabs/featuretools/issues/130
"notifications" : (notifications_pd, "notification_id"),
"transactions": (transactions_pd, 'transaction_id'),
}
In [64]:
relationships = [("users", "user_id", "transactions", "user_id"),
("users", "user_id", "notifications", "user_id"),
("users", "user_id", "devices", "user_id")]
Run deep feature synthesis¶
In [65]:
feature_matrix_users, features_defs = ft.dfs(entities=entities,
relationships=relationships,
target_entity="users", # we wish to profile users, so it's target
# n_jobs=multiprocessing.cpu_count()-1,
# I'm getting "zmq.error.ZMQError: Too many open files" - MacOS issue?
verbose=True)
In [66]:
# features_defs
In [67]:
feature_matrix_users
Out[67]:
Questions¶
1. Define a target metric to measure user engagement.¶
How would you define an engaged vs. unengaged user?¶
Please provide the business justification and associated visualisations / rationale in choosing your definition of engagement.
Propose type of segments¶
Using my domain knowledge and an intuition (as an active Revolut user), I'd like to segment users into 3 segments:
- Super-commited users
Users, who make transactions in their home country. They're not afraid to use Revolut on their daily basis, even for local ATMs and online purchases.
- Engaged users
Users, who only make transactions abroad. Their main incentive is to save money on currency exchange spread. They're mostly people with non-premium subscriptions.
- Unengaged users
Users, who don't make much tranactions, even when they're abroad. Low activity, low responsivness to marketing.
It's practically impossible (with current state of data) to detect whenever the user goes abroad and doesn't use his/her Revolut actively there. While it's very easy to count number of transactions per country, this doesn't answer the question if the user went abroad (or not). Because of those facts, we're going to propose some other metric. Recently, customer loyalty has been measured by a metric called "RFV". Let's modify it by adding another dimension - response to "push" notifications. Such RFV-R metric should be validated against 3 segments defined above (super-commited, engaged, unengaged) by adjusting importance weights.
References:
- Recency – when did they last make a transaction?
- Frequency – how often do they make transactions?
- Volume – how many transactions have they made?
- Responsivity - how long it takes to make a transaction since last notification?
Calculate RFV-R¶
Because calculating RFV score is computational expensive, we're going to calculate this metric only at the last timepoint available in the data. Normally, RFV should be computed on a transactional level and updated after every transaction.
In [68]:
f'Profiling {len(users_pd)} users.. it may take some while'
Out[68]:
In [69]:
transactions_pd['year-month'] = transactions_pd['created_date'].apply(lambda x: f'{x.year}-{x.month}')
In [70]:
# we're using a groupby approach
recency = transactions_pd.groupby('user_id').agg({'created_date': 'max'}) # recency
In [71]:
# recency
In [72]:
frequency = transactions_pd.groupby(['user_id', 'year-month'])['transaction_id'].count() # frequency
In [73]:
# frequency
In [74]:
volume = transactions_pd.groupby('user_id')['transaction_id'].count() # volume
In [75]:
abs_volume = transactions_pd.groupby('user_id')['amount_usd'].sum()
In [76]:
# volume
In [77]:
# responsivity
last_notification = notifications_pd.groupby('user_id').agg({'created_date': 'max'})
In [78]:
users_enriched_pd = users_pd.copy()
In [79]:
users_enriched_pd['recency'] = users_enriched_pd['user_id'].apply(lambda x: (max_date_dt -
recency.loc[x]['created_date']).days
if x in recency.index else None)
In [80]:
users_enriched_pd['frequency'] = users_enriched_pd['user_id'].apply(lambda x: frequency.loc[x].values.mean()
if x in frequency.index else None)
In [81]:
users_enriched_pd['volume'] = users_enriched_pd['user_id'].apply(lambda x: volume.loc[x]
if x in frequency.index else None)
In [82]:
users_enriched_pd['responsivity'] = users_enriched_pd['user_id'].apply(lambda x: (recency.loc[x]['created_date'] -
last_notification.loc[x]['created_date']).days
if (x in recency.index) and (x in last_notification.index) else None)
In [83]:
users_enriched_pd['responsivity'] = users_enriched_pd['responsivity'].apply(lambda x: None if x < 0 else x)
# it may happen that a notification was sent recently, and the user hasn't done anything
Analyze type of distributions¶
In [84]:
sns.pairplot(users_enriched_pd[['recency', 'frequency', 'volume', 'responsivity', 'plan']].fillna(0), hue='plan')
Out[84]:
Pseudo-label¶
Using logic from above, we're trying to categorize users using simple heuristics and intuition. In a result, we'd like to fit a curve which would normalize our RFV-R score to such in range (0,1).
In [85]:
# engaged: super-commited, min 3 countries, premium users, metal users
In [86]:
country_stats = transactions_pd.groupby(['user_id', 'ea_merchant_country'])['ea_merchant_country'].count()
In [87]:
# country_stats
In [88]:
users_enriched_pd['pseudo-label'] = users_enriched_pd.apply(lambda x: (x['plan'] in ['PREMIUM',
'PREMIUM_FREE',
'METAL',
'METAL_FREE']) or
(len(country_stats.loc[x['user_id']]) > 4 if x['user_id'] in country_stats.index else False),
axis=1)
In [89]:
# unengaged: less than 3 countries
In [90]:
users_enriched_pd['pseudo-label'].value_counts()
Out[90]:
In [91]:
users_enriched_pd['pseudo-label'] = users_enriched_pd['pseudo-label'].astype(int)
In [92]:
y = users_enriched_pd['pseudo-label'].values
In [93]:
rfv_columns = ['recency', 'frequency', 'volume', 'responsivity']
In [94]:
X = users_enriched_pd[rfv_columns].fillna(0).values
In [95]:
y
Out[95]:
In [96]:
clf = LogisticRegression(solver='lbfgs').fit(X, y)
In [97]:
clf.score(X, y)
Out[97]:
In [98]:
coefficients = pd.DataFrame({"Feature": rfv_columns, "Coefficients": np.transpose(clf.coef_[0])})
In [99]:
coefficients
Out[99]:
In [100]:
clf.classes_
Out[100]:
In [101]:
users_enriched_pd['rfv_score'] = users_enriched_pd.apply(lambda x:
clf.predict_proba(np.nan_to_num(np.array([x['recency'],
x['frequency'],
x['volume'],
x['responsivity']]).reshape(1, -1)))[0][1],
axis=1)
In [102]:
users_enriched_pd['rfv_segment'] = users_enriched_pd.apply(lambda x:
clf.predict(np.nan_to_num(np.array([x['recency'],
x['frequency'],
x['volume'],
x['responsivity']]).reshape(1, -1)))[0],
axis=1)
In [103]:
users_enriched_pd['rfv_score'].hist()
Out[103]:
In [104]:
users_enriched_pd.columns
Out[104]:
In [105]:
sns.pairplot(users_enriched_pd[['recency', 'frequency', 'rfv_segment', 'pseudo-label',
'volume', 'responsivity']].fillna(0),
hue='rfv_segment', diag_kind='hist')
Out[105]:
Check for Pareto principle¶
In [106]:
# 20% of commited users should be responsibe for 80% of transactions
In [107]:
sorted_users_enriched = users_enriched_pd.sort_values('rfv_score', ascending=False)
In [108]:
top_20 = sorted_users_enriched.head(n=int(0.2 * len(sorted_users_enriched)))
In [109]:
tail_80 = sorted_users_enriched.tail(n=int(0.8 * len(sorted_users_enriched)))
In [110]:
top_20['total_usd'] = top_20['user_id'].apply(lambda x: abs_volume.loc[x] if x in abs_volume.index else None)
In [111]:
tail_80['total_usd'] = tail_80['user_id'].apply(lambda x: abs_volume.loc[x] if x in abs_volume.index else None)
In [112]:
top_amount_usd = top_20['total_usd'].values.sum()
top_no_of_transactions = top_20['volume'].values.sum()
In [113]:
tail_amount_usd = tail_80['total_usd'].fillna(0).values.sum()
tail_no_of_transactions = tail_80['volume'].fillna(0).values.sum()
In [114]:
top_amount_usd / (top_amount_usd+tail_amount_usd)
Out[114]:
In [115]:
top_no_of_transactions / (top_no_of_transactions+tail_no_of_transactions)
Out[115]:
Calculate Gini coefficient¶
In [116]:
# TODO: calculate inequalities between segment of engaged users and not-engaged users
2. Using your logic from above, build a model (heuristic/statistical/ML) to classify 'engaged' and 'unengaged' users¶
Note that features which are directly correlated with your target metric could lead to overfitting¶
In [117]:
users_enriched_pd = users_enriched_pd.drop(['rfv_score', 'pseudo-label'], axis=1)
In [118]:
labels = users_enriched_pd['rfv_segment'].values
In [119]:
users_enriched_pd = users_enriched_pd.drop(['rfv_segment'], axis=1)
In [120]:
# rerun deep-feature-synthesis
entities2 = {"users" : (users_enriched_pd, "user_id"),
"devices" : (devices_pd, "device_id"),
"notifications" : (notifications_pd, "notification_id"),
"transactions": (transactions_pd, 'transaction_id'),}
In [121]:
feature_matrix_users, features_defs = ft.dfs(entities=entities2,
relationships=relationships,
target_entity="users",
verbose=True)
In [122]:
# feature_matrix_users
In [123]:
feature_matrix_users_cat = pd.get_dummies(feature_matrix_users)
In [124]:
# feature_matrix_users_cat
feature_matrix_users_cat_nonulls = feature_matrix_users_cat.fillna(0)
In [125]:
X_train, X_test, y_train, y_test = train_test_split(feature_matrix_users_cat_nonulls, labels,
test_size=0.20,
random_state=42)
In [126]:
rf = RandomForestClassifier(random_state=42)
gdb = GradientBoostingClassifier(random_state=42)
In [127]:
rf.fit(X_train, y_train)
Out[127]:
In [128]:
gdb.fit(X_train, y_train) # takes 2-3 minutes to compute
Out[128]:
In [129]:
rf_preds = rf.predict(X_test)
In [130]:
gdb_preds = gdb.predict(X_test)
In [131]:
target_names = ['Not-engaged', 'Engaged user']
rf.classes_, target_names
Out[131]:
In [132]:
print(classification_report(y_test, rf_preds, target_names=target_names))
In [133]:
print(classification_report(y_test, gdb_preds, target_names=target_names))
In [134]:
explainer = shap.TreeExplainer(rf)
In [ ]:
shap_values = explainer.shap_values(X_train, check_additivity=False)
In [ ]:
shap.summary_plot(shap_values, feature_matrix_users_cat_nonulls)
In [ ]:
explainer = shap.TreeExplainer(gdb)
In [ ]:
shap_values = explainer.shap_values(feature_matrix_users_cat_nonulls)
In [ ]:
shap.summary_plot(shap_values, feature_matrix_users_cat_nonulls)
3. Let’s assume an unengaged user is a churned user. Now suppose we use your model to identify unengaged users and implement some business actions try to convert them to engaged users (commonly known as reducing churn)¶
How would you set up a test/experiment to check whether we are actually reducing churn?¶
We would take one action with one group of customers, and take a different action (or often not action at all) with a control group, and then compare the results. It means selection of participants and assignment of them to "treatment" and control conditions, preferably using a random procedure (another idea, is to use stratified or systematic sampling).
The final action only would be applied to the entire population of customers if it is a success.
The outcomes must be simple to analyze, and the data should be easily interpreted. We must monitor the research situation to ensure that there are no differences between the treatment and control conditions other than the "intervention".
Measurement of selected outcomes for both groups combined together with statistical analysis to determine if the groups differ on those dependent variable measures.
What metrics and techniques would you use to assess the impact of the business action?¶
Outcome metric - 'Loss/gain of churn rate'
Intermediate metric - change of RFV score
Comparision methods - One-way ANOVA test